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ABSTRACT 

The paper describes an interactive valuation process 
based on E. Brown* s transactional evaluation model, and its 
application to evaluating programs serving children with severe and 
multiple disabilities or other low incidence populations. The model 
emphasizes th* first part of the assessment process, determining 
which questions to ask and what kind of information can provide 
suitable answers. Individualization of the evaluation design is 
accomplished through structured interactions with interested parties. 
Such meetings allow program staff, students, parents, administrators, 
advocates, and others to participate in the evaluation design. 
Subsequent meetings with these same parties allow for additional 
input and improve prospects for implementing change, (Author/CL) 
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Abstract. 



This paper describes an interactive evaluation process based on 
Brown's (19 '8) transactional evaluation aodel. The process is well suited 
to evaluation of programs serving children with severe and multiple 
disabilities or other low-incidence populations. This model places 
emphasis on the first part of the assessment process, determining which 
questions to ask and what kind of information can provide suitable answers. 
Individualization of the evaluation design is accomplished through 
structured interactions with interested parties.. These meetings allow 
program staff, students, parents, administrators, advocates, and other 
interested parties to participate in the evaluation design. Subsequent, 
meetings with these same parties allow for additional input before 
recommendations are developed and improve the prospect of implementation of 
change . 
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Interactive Evaluation of Educatio n al Prog ram, Serving 
Students with Severe Handicap s 

Brown (1978) proposed a transactional .odel of school evaluation which 
used structured interactions with interested partis to establish the 
content of an evaluation. This model of evaluation has been applied 
successfully to evaluate of school program serving students with severe 
and .ultiple handicaps This paper discusses the evaluation process., with 
emphasis on the develops of evaluation questions and standards. Since 
some changes have been made in Brown's model, the ter, "interactive" is 
used (rather than "transactional » ) to minimize confusion between his 
original model and this application. 

During the 1980's, increasing emphasis on ,ff ici ent utilization of 
lifted and sometimes shrinking resources has increased the demand for 
-uungfol evaluation of educational services. The sma ll size, 

individualized curricula., and student-centered objectives associated with 
programs designed to serve students with sev ere and multiple disabilities 
require an evaluation process tailored to the specific program An 
interactional evaluation model provides a method of fitting the evaluation 
to specific program characteristics and Reals. 

Interactive evaluation uses a number of traditional methods associated 
with various evaluation models to answer relent questions, but differs 
from traditional models in its method of formulating these questions. It 
utilizes structured interactions between the evaluator or evaluation team 
and key individuals (e.g.., administrators., program staff., parents., and 
students) along with empirically verified effective instructional 
parameters to develop a plan for evaluation. 
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The term,, "program evaluation," is in itself potentially misleading. 
It implies that an entire program is assessed in some way, but programs, 
like people, are complex individuals, and unless evaluation is limited to a 
specific subset of characteristics, the process often becomes unmanageable 
and the product often becomes uninterpretable . Therefore,, one of the first 
and most important tasks in any evaluation is to determine exactly which 
characteristics to evaluate,, or to put it another „ay, : exactly which 
questions to ask. 

Almost any legitimate question may be asked in some form, but each 
should be carefully considered to ensure (1) that there is agreement about 
the importance of the question, and (2) that information used to answer the 
question can be identified,, and (3) that the required information is 
accessible. For example,, student outcome data is considered important in 
most evaluations. Most often this is identified as some specific measures 
of studsnt progress over time. These measures are typically available as 
formative or summative data from on-going programs. m some cases,, 
however,, very little data is available to use. If at least a single,, 
suitable assessment of student performance is available from an earlier, 
identifiable point in time, it may be used as a pretest measure and 
compared with a postest measure taken during the evaluation to measure 
-ogress. Although less helpful than formative data (taken daily during 
each program),, this can provide a reasonable measure of progress. Without 
such an earlier measure to serve as a pretest, however, a meaningful 
assessment of student progress becomes impossible, unless the evaluation 
period is several months long. In such cases, the evaluation team may need 



9 

ERIC 



5 



to determine that they cannot answer this question and consider options 
(e.g.,, extending the duration of evaluation activities so the question can 
be answered or restricting the evaluation to other questions). Such 
unanswerable questions may suggest other important evaluation questions (in 
this case, 'Are on-going student progress measures adequate?"). 
DETERMINING EVALUATION n nr^Tinvc 

The content of evaluations say be determined by a number of different 
sources Sometimes, evaluators independently decide what to assess,, based 
o, what they consider to be of interest or their knowledge of current 
reseaich and practice. Sometimes, evaluation funding sources clearly 
specify what must be evaluated. Often, administrators, program staff,, 
Parents, advocates,, service consumers. an d . ther interested parties would 
like to have particular concerns addressed in evaluation. 

Interactive evaluation places mo re emphasis on determining content 
than traditional evaluation on content selection. This requires meetings 
with all interested parties to establish an evaluation plan prior to 
subsequent evaluation activities,, and continued meetings with these groups 
at critical stages during the evaluation. A basic sequential outline of 
activities (emphasizing interactions with interested parties) might 
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include . 



(A) Develop evaluation timeline 

(B) Determine intererted parties 
( c) Meet with int ere sted parties 

1 ■ hyplain process 

2 Determine questions and standards 
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(D) Circulate questions and standards for review and approval 

(E) Collect evaluation data 

(F) Draft preliminary findings 

(G) Circulate finding* t 3 interested parties 
(H * Meet with interested p artis 

1 . Discuss findings 

2. Consider revisions 

3 Consider possible recommendat j0 ns 
(I) Revise findings if appropriate 
Uj Write preliminary recommendations 
(KJ Circulate to interested parties 
(L) Meet with interested parties 

1. Discuss recommendations 

2. Consider revisions 

(M) Write final recommendations 
( N * Meet with interested p .rt^c 

1 ■ Present final repurt 

2. Discuss implementation 
(0) Circulate final report 

Since his requires meetings with interested parties at at least four 
stages. so,ne extra time may be needed, and a detailed timeline is essential 
to coordinate activities in a reasonable period of time. Brown (1978) 
suggests a good method for breaking down the evaluation process into tasks, 
activities, and transactions. This .ethod is illustrated in the example 
she™ in Flgure T1 „ es can be ass . gned |o transactl<>ns ^ co>i)jete 

timeline process 
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Insert Figure 1 about here 



Determining who the interested parties are usually simple, slnc e 
typically these people already interact with the program. Asking the 
program staff. administrators and parents who are immediately accessible 
about other interested parties may help identify additional sources. Once 
these parties have been identified, the evaluation team must decide whether 
to try to meet with the, together as one large group or in several smaller 
sessions Meeting as one large group avoids having to integrate the 
results of several smaller meetings, reduces the number of meetings 
required, and encourages communications between interest groups, but often 
smaller meetings are better because ideal times for various groups (e.g.,. 
Parents and teachers) are different,, better participation occurs in smaller 
groups, and shorter meetings draw better attendance. 

Once participants are identified and meetings begin,, an evaluation 
Plan can be developed As each evaluation question is deve^d. standards 
for a wering the questions., sources of dat ,. and instruments for 
collecting da.a mU st be identified. Figure 2 illustrates a sample plan. 
It is important to note that while this information must be specific enough 
to provide direction, it is not as specific as the actual data collection 
instruments . 



Insert Figure 2 about here 



FINDING THE ANSWERS 

Usually, the best data involves direct .easi.re.ent of behavior. For 
example, teacher or student instructional ti.e can be observed directly. 
Some questions, however, cannot be answered easily through dire ct 
observation and are better measured through survey data. it may also be 
important to assess the social validity of empirical data. For example, 
Parental satisfaction with the program is best assessed by interview or 
survey, and although student progress should be empirically measured, 
teacher and parent perceptions of student progress may also be of interest 
as a social validation measure. Figure III provides an example of a survey 
instrument. Since some elements of a number of different evaluation 



Insert Figure 3 about here 



Questions are addressed through each survey instrument,, these instruments 
should not be selected or developed until the entire evaluation plan is 
completed . 

Once the evaluation plan and instruments are fully developed the 
evaluation team can use whatever means methods and models that seem most 
appropriate. Discrepancy evaluation is a popular and useful approach which 
looks at the match or lack of match between the program and standards 
(Steinmetz. 1983),, but there are many useful strategies for using data to 
answer specific evaluation questions. An electic approach allows the 
evaluators to use the best evaluation model for each question to be 
answered 
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The additional interactions that occur after the preliminary findings 
allow interested parties to participate in making recomuendat ions . Their 
participation at this stage nay result in better recommendations, and also 
helps ensure that greater participation in the implementation of 
recommendations that follows 
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Figure 1 



Breakdown 


of Evaluatior Tasks [Excerpt] 




TASK: 


Design Evaluation 


|No. i 


SCOPE: 


Determine Evaluation Question* 




PURPOSE: 


To identify questions that reflect the concerns of 
administration, staff, parents and advocates 





ACTIVITIES: 

Literature Review 

for evaluation Questions 



Interaction w i tn interested 
parties for genera ;i,ig 
evaluation cuesfons 



Icmrrumsation with 
acvocacy groups for 
generating evaluation 
questions 



TRANSACTIONS: 

1.0 Review policies and guidelines 

1.1 Review Alberta Education program 
poHcies and guide. ines 

1.2 Review ECSD guidelines and oolicies 
for D.H. program 

2.0 Review current educational literature 

3.0 Meet with interested parties 

3.1 Meet with ECSD administrators 

3.2 Meet with evaluation steering 
coram ttee 

3.3 tfeet with parents of students in 
D.H. program 

3.4 ^eet witn D.H. progran staff 

4.0 Communication with advocacy 
groups 

4.1 Send draft evaluation plan to 
advocacy groups witn *equest, 
for suggestions 

4 ''' Incorporate suggestions into 
evaluation clan 



(Sojsey, 1985; after format of Brown, 1978) 
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F igure 2 

Excerpt from Evaluation plan 

EVALUATION PLAN 



QUESTION 



STANDARD 



SOURCE 



INSTRUMENT 



1.0 Does the program 
provide a social- 
emotional climate 
supportive of learning 
development of sel f- 
esteem? 



1.1 Teacher attitudes a 1 "-' 
expectations are 
posi tive. : 

1.2 Non-program school 
staff attitudes are 
posi tive. 

1 .3 Non-handicapped 
students attitudes are 
supportive. 

1.4 High ratios of reinforce- 
ment and error correction 
are maintained.. 

1.5 Steff-pupil interaction 
rate high when compared 
to sta f f-staff 
interaction. 

1.6 Staff maintain physical 
proximity and contact 
with pupils. 

1.7 Parents report children 
"pnjoy" school program. 

1 .8 Teaching staff rate 
interactive components 
of job as desirable. 

1 .9 Opportunities for 
social interaction are 
maximired. 



teachers 



staff 



non-handicapped 
students 



direct observation 
direct observation 

direct observation 
parents 

teachers and aides 



teacher survey 

staff survey 
interview notes 



videotape and 
evaluators notes 



videotape ard 
evaluators notes 



videotape and 
evaluators rotes 



videotape and 
evaluators notes 
videotape and 
evaluators notes 



(Sobsey, 1985) 
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Figure 2 (continued) 



EVALUATION PLAN 
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QUESTION 



STANDARD 



SOURCE 



INSTRUMENT 



18.0 Are integration and 

nor ira 1 ization occuring 
to the greatest extent 
possible 7 



18.1 School ! ncludes non- 
handicapped peers. 

18.2 School allows students 
in depenu3nt disabled 
program to attend 
part-day programs in 
other categorized 
classrooms. 

18.3 Students in dependent 
disabled program have 
established pathways 
to graduate into other 
programs in school 
and/or system* 

18.4 Setting is age 
appropriate. 

18.5 Activities are age 
appropriate. 

18.6 Non-academic 
activities are 
integrating students. 



Administrators 
Parents 
Teachers 
Program staff 
Direct observation 



Request letters 
Parent survey 
Teacher survey 
Evaluators notes 



(Sobsey, 1985) 
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Figure 3 




Excerpt from Survey Instrument 






TEACHER AND AIDE QUESTIONNAIRE 




l. 


What is your positior? 

a) teacher airia r*\ „ *. l. _ ^ . _ 
a; vca^nef q} aiae c; other (please specify; 




2. 


How long have your worked in this program? 
years months 




3. 


hp nH i rinc h f dld y° u have with students witn severe and multiple 
handicaps before coming to work in this program? 




4, 


What experience did you have in other areas of special education befo-° 

rorn i no t n work in v.„ „ ^ , _ o wt 1 ^ - 
^.uii.jiiy lu w^rs in cms program; 




*> 


25rk fnTh7s en p "c d r^ y0U ^ eduC3ti0n bef ° re COnl1n 9 t0 




6. 


What training did you rocp^ve which provided specific preparation for 
ycu 5cei°ve It* stuGGnts w1th severe and multiple handicaps? When did 




7 . 


What other special education tra-mng did you receive? When did vcu 
receive it? 




8, 


What other general education training did ycu -eceive? When did vou 
recei ve i t? v 




(Sobsey, 1985) 
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Figure 3 (continued) 



9. How aaequa^ly do you feel your training prepared ycu to work with 
students with severe and multiple handicaps? 



nJTTy weTi adequately inadequately mostly completely 

prepared prepared prepared prepared unprepared unprepared 

10. How important are individual program plans for the education of your 
students? J 



extremely rarely sometimes frequently extremely 

unimportant useful useful useful * important 

11. How important are periodic assessments of overall performance for your 

students? - J 



extremeiy rarely sometimes frequently extreme] v 

unimporcant useful useful useful important 

12, Hew important is daily collection of data on each prcaram for vour 

students? " 



extreme \y rarely sometimes frequently extremely 

unimportant useful useful useful important 

13. For each of the living situation categories listed below, write in the 
number of D.N. students that you currently work with that, vou think 
might eventually live in that category. 

aj total independent 

bj semi-supervised group home/ccmpeti ti ve employment 

c) supervised group home/sheltered employment 

dj total grouo home or parental supervision 

e) institutional care 

U. a) How much integration of activities occurs between students in the 
D.H. program and their ron-nandiCcDced pee^s. 



no 
iptocra 



(So! 



on 
sey. 



ver- 
iir*] 

1985* J 



little 



some 



a lot 



integration 
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